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Growing demands for accountability and useful 
measurement of educational outcomes have contributed to increasing 
efforts to develop or refine more valid and reliable methods of 
educational assessment. More states are designing high-quality 
indicator systems for evaluation at the school, district, and state 
levels. Seven recommendations derived from research and practice for 
developing practical indicator systems are presented in this 
educational policy bulletin, with a focus on defining objectives and 
reporting data. The challenge is to develop indicator systems that 
are useful to and used by educational stakeholders. (LMI) 
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Developing Useful Educational Indicator Systems 

When the Office of Educational Research and Improvement (OERI) released its report, Creating Responsible and Respon- 
sive Accountability Systems, a year ago, Study Group Chairman Terry Peterson called accountability a "very hot" issue. It 
still is. Demands for accountability and for useful ways of measuring it are coming from governors, legislators, business 
persons, educators, and parents. "There is a change in the public's expectation of what they want schools to accomplish," 
says Louisiana Superintendent Wilmer Cody. 

Currently, many states are working to develop or refine "report cards," update or create more valid and reliable tests, and 
design measures of higher-order competencies. In short, states are designing high-quality indicator systems that will be 
useful to policymakers, educators, and the public in assessing the condition of education at the school, district, or statewide 
level. 
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Developing Useful Indicator Systems 

"The key to developing indicators is deciding what 
you want your indicators to do for the state," says 
Ramsay Seld.en, Director of the State Assessment 
Center of the Council of Chief State School Offi- 
cers. Indicators must be selected, collected, and 
reported according to a broad array of policy needs 
and the public's concerns or expectations. 

Setting up an indicator system that is both fair and 
politically useful is no easy assignment. "Useful- 
ness," writes Shavelson et al. (1987), 

depends on much more than just the dissemi- 
nation of indicator data. It also hinges on 
factors such as the indicators contained in 
the system, how they are conceptualized and 
measured, the ievel at which they can be ag- 
gregated, and the way in which they are ana- 
lyzed and reported (p. 22). 

Creating useful indicators means making difficult 
tradeoffs, admits Terry Peterson. Sound accounta- 
bility systems must weigh tradeoffs between a 
state's goals for assessment and/or improvement 
and the resources available for such a system; the 
need for state accountability versus local auton- 
omy; the need for statewide comparability against 
the need for local ownership; the utility of collect- 
ing and analyzing new data against the burden in 
time, effort, and paperwork to do so (OERI, 1988). 



The challenge, then, as David (1988) frames it is 
twofold: (1) to create indicators that policy- 
makers, teachers, and the public will use and (2) 
to create conditions that enable stakeholders to 
use indicators in their own planning and decision- 
making. 

To meet that challenge, educational decision- 
makers can draw on the following lessons from 
literature and the experiences of other states to 
develop indicators that are useful to and used by 
educational stakeholders: 

1 . Engage stakeholders in defining goals. 

2. Be realistic about the role of indicators 

3. Develop multiple indicators to reduce unin- 
tended effects. 

4. Consider the resources available for operat- 
ing the system. 

5. Report data so that they allow for fair com- 
parisons. 

6. Report data so that they are understandable 
and useful to a variety of audiences and stake- 
holders. 

7. Assess and revise the system as necessary. 
Lessons from Research & Practice 

1. Engage stakeholders in defining goals. "In 

the nation today, the imperative for accountability 
is no longer a mandate from the top, but is 
focused on the kids," according to William 



Southwest Educational Development Laboratory 

211 E. Seventh Street, Austin, Texas 78701 (512) 476-6861 



Youngblood, President of the South Carolina 
Business Roundtable. "It is not a school prob- 
lem," he continues, "it is an economic develop- 
ment problem for the community." Local deci- 
sionmakers create a climate for developing useful 
indicators when they invite school or district staff 
along with the community at large to suggest what 
should be measured and how. 

Youngblood says designers of an indicator system 
must ask themselves three questions: 

• Have you adapted your vision to what you 
want the schools to accomplish? 

• Do you know where you want the system to 
be to meet the demands in a global economy? 

• Do you have ways of knowing whether the 
system is on track? 

South Carolina developed its goals for an indica- 
tors system by achieving consensus through 
meetings with business persons, parents, and 
teachers around the state. A blue ribbon commit- 
tee established the most important goals, which 
were then written into the state's Education Im- 
provement Act. 

2. Be realistic about the role of indicators. Edu- 
cational indicators are single or composite statis- 
tics that reveal something about the performance 
of an educational system. They are tools that can 
help state and local policymakers make decisions 
to improve the education system. They may be 
used simply to report information or they may be 
used to monitor school performance. 

Indicators can measure the dimensions of the 
school system that are related to performance. 
They can flag or begin to diagnose problems of 
schools or districts and they can suggest potential 
causes of such difficulties. They have limitations, 
however. Indicators cannot meet all data needs, 
portray the education system perfectly, or show 
cause and effect relationships. 

An indicator system measures individual compo- 
nents of the educational system, but it also tells 
audiences how the components are interrelated, or 
how they work together to bring about desired 
outcomes. If indicators are to be used to examine 
relationships among the components of an educa- 
tional system, they ought to be generated from a 



model that depicts how the system actually works. 
Generally, such a model includes inputs, proc- 
esses, and outcomes (Oakes, 1936; Shavelson, 
McDonnell, Oakes, & Carey, 1987). 

In Louisiana, educational stakeholders have been 
researching "input-process-outcome" indicators 
for the state's progress profiles. They are in the 
process of examining indicators and have recom- 
mended the following: 

• Student outcomes-results on state criterion- 
and norm-referenced tests, dropout statistics, 
student attendance data, ACT/SAT results, 
graduation statistics 

• Demographics-SES of students, relative 
wealth of district 

• Other student data-suspension/expulsion, 
participation in activities outside of school, 
student attitudes, socialization and interper- 
sonal relations information 

• Input measures-faculty/staff characteristics, 
class size, fiscal effort for education, adminis- 
trative leadership academy participation, 
teacher evaluation information 

• School process measures-information about: 
school climate, educational leadership, parent/ 
community involvement/ support, effective 
teaching, coverage of curricular content 

• Optional indicators-national merit semi- 
finalist statistics, advanced placement classes, 
SAT school-level results, statistics on availa- 
bility and participation in preschool and alter- 
native programs (Louisiana State Department 
of Education, 1989). 

3. Develop multiple indicators to reduce unin- 
tended effects. Unintended consequences of an 
indicator system may corrupt data and make the 
system less valid or reliable. "When you begin to 
measure something, you may bring into being 
some unwanted effects," says Jane David, educa- 
tional consultant and author of Improving Educa- 
tion with Locally-Developed Indicators. 

Accountability systems might actually stifle crea- 
tivity or school improvement by causing a nar- 
rowing of curriculum offerings in some states. 
"What you measure is what schools will spend 
time on," she cautions. The question is not, "How 
do you get rid of standardized test scores as an 
indicator?" but, "How do you minimize potential 
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unwanted effects?" David suggests that one way 
is to give districts and schools some choices in 
how they measure achievement. Standardized test 
scores are a relatively easy, inexpensive way to 
measure performance, but districts should be able 
to use other measures of achievement if they can. 

4. Consider the resources available for operat- 
ing the system. Indicator systems require equip- 
ment, funding, time, and human resources for 
collecting, analyzing, and reporting data. 
Whether the data are analyzed and reported by 
school or by district, much of the burden of data 
collection is on teachers. The effort and time 
required to collect data must be weighed against 
potential losses in instructional time. For ex- 
ample, the usefulness of indicator data to class- 
room teachers is limited largely because of the 
expense and burden entailed in collecting, analyz- 
ing, and reporting data that provide teachers with 
detailed feedback about how their instructional 
strategies affect classroom performance (Porter, 
1988). 

A state department should have some analytical 
capacity, too. Or, as the California State Depart- 
ment of Education has done, it may contract with 
an external research agency or university to 
analyze indicator data. 

Lynn Moak, Deputy Commissioner for Research 
and Information at the Texas Education Agency, 
advises that, whenever possible, indicator systems 
should take advantage of existing regional net- 
works and computer technology to control re- 
source burdens. In Texas, for example, regional 
educational service centers are responsible for 
editing raw data after they are collected. 

5. Report data so that they allow for fair com- 
parisons. Local school administrators and poli- 
cymakers share concern about the ability of an 
indicator system to allow fair comparisons among 
schools. That is, how can a system fairly compare 
the performance of schools in a well-funded, 
suburban district with the performance of remote, 
resource-bound schools populated by poor or 
disadvantaged students? What should be the 
features of similarity: per pupil spending, free 
lunch enrollment, parents' educational attain- 
ment? Providing for some basis of comparability 
creates a system that "recognizes social context 
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but holds high expectations for all students" 
(OERI, 1989). 

A system can provide comparability by reporting 
information about the context of schools or dis- 
tricts, asserts Ellen Still, Director of Research for 
the Senate in South Carolina. South Carolina and 
California both report schools' performance 
within "comparison bands" of demographic and 
socioeconomic data. Each school's expected 
performance is figured on the basis of previous 
test scores, and is compared to schools with 
similar student characteristics within their band. 

6. Report data so that they are understandable 
and useful to a variety of audiences and stake- 
holders. "If working properly, accountability 
ought to give parents and educators the informa- 
tion they need to request more changes," says 
Chris Pipho of the Education Commission of the 
States. The best way to give them such informa- 
tion is to tailor the report to the audiences. "Keep 
your audience in mind," says Ellen Still. South 
Carolina produces three different reports for its 
audiences: a school performance report for 
principals and faculty, an "impaired district" 
report for district administrators, and a statewide 
assessment report with a four-page pamphlet for 
public readers. She also advises staff who pro- 
duce the report to offer technical assistance on 
reading and understanding the demographics and 
the characteristics of students. "Most people 
don't know about the characteristics of students. 
Spend some time traveling through the state to tell 
them." 

7. Assess and revise the system as necessary. 
"No system starts out perfect," asserts Still. Con- 
sider the system to be in a "phase-in" period, she 
adds, and start with the measures that are avail- 
able. Texas has gradually added to the data that 
the state collects through its Public Education 
Information Management System (PEIMS). "You 
also have to be able to change indicators if they 
no longer measure what is important to your 
goals," says Bill Youngblood. And, he cautions, 
you have to be patient. Most reform efforts, like 
new businesses, fail. "Keep in mind that you're in 
it for the long run." 
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